1. Field of the Invention
The present invention relates to an intermediate value pixel judgment method for judging whether a reference pixel, which is used for interpolating an interpolation pixel between pixels included in an image represented by image data, is an intermediate value pixel when the image is enlarged or reduced, for example. The present invention also relates to an image interpolation method for interpolating the interpolation pixel between the pixels included in the image. The present invention also relates to a method for adjusting an edge, which is detected to interpolate an interpolation pixel between the pixels. The present invention also relates to an edge detection method for detecting an edge located between pixels in an image. The present invention relates to these image processing methods and apparatuses, and programs for causing a computer to execute these methods.
2. Description of the Related Art
Conventionally, a digital image, which is obtained by photoelectrically reading out a photograph image, which is recorded on a photography film such as a negative film and a color reversible film, with a read-out device such as a scanner, and an image represented by image data, which is obtained by taking a photograph of a subject with a digital photography device such as a digital still camera (DSC) and a cellular phone to which a camera is attached, are enlarged or reduced so that the size of the images conforms to the size of a reproduction device such as a monitor for reproducing the images. For example, a service is provided for enlarging or reducing the image data according to the size of a liquid crystal monitor of a cellular phone at a destination when an image data, which is obtained by taking a photograph of a subject with a cellular phone to which a camera is attached, is sent to the cellular phone at the destination as an e-mail.
The image data is enlarged or reduced by interpolating a new pixel (hereinafter, referred to as an interpolation pixel), based on an enlargement/reduction ratio, between pixels included in an image, which is represented by image data. There are various known interpolation methods such as a linear interpolation method, a Gaussian filter method, a nearest neighbor method, a bilinear method, and a bicubic method.
However, if the image data is enlarged or reduced by using a single method, there is a problem that a blur or a “shaggy edge” or shagginess is generated in an edge part included in the image. Therefore, a method has been proposed for detecting an edge component included in an image and performing a different interpolation operation for an edge part and for a non-edge part from each other (Japanese Unexamined Patent Publication No. 2002-319020).
A Sobel filter or a Laplacian filter is generally used for detecting an edge in the image by the method disclosed in Japanese Unexamined Patent Publication No. 2002-319020 or like. The Sobel filter and the Laplacian filter are odd-tap filters, of which shortest length is 3 taps. An edge can be detected by judging whether a target pixel is an edge by performing filtering processing.
Meanwhile, when an image is enlarged or reduced, an interpolation pixel is interpolated between pixels. Therefore, it is required to judge whether there is an edge between the pixels instead of judging whether there is an edge in a pixel in the image. When edge detection processing is performed by using the odd-tap filter as described above, it is possible to judge whether the target pixel, itself, in the image is an edge. However, it is impossible to judge whether there is an edge between the pixels.
As the simplest method for judging whether there is an edge between pixels, there is a method for performing filtering processing on two adjacent pixels in an image, for example, by using a difference filter, which is illustrated in FIG. 5. According to this method, a difference in pixel value between the two pixels is obtained, and if the absolute value of the difference is equal to or larger than a predetermined threshold value, it is judged that there is an edge between the two adjacent pixels, for example. In the following descriptions on the present invention, the edge detection method as described above is referred to as a first edge detection method.
Further, a method for judging whether there is an edge between two pixels, which are located at the middle among four serially adjacent pixels, based on the pixel values of the four serially adjacent pixels instead of the pixel values of the two adjacent pixels may be also used. Specifically, for detecting whether there is an edge between two pixels G2 and G3, which are located at the middle among four serially adjacent pixels G1, G2, G3 and G4, which are illustrated in FIG. 4, first, filtering processing is performed on three pixel pairs (G1, G2), (G2, G3), and (G3, G4), each including mutually adjacent two pixels by using a difference filter. Accordingly, the difference in pixel value between each pair is calculated as primary differences d1, d2, and d3. Then, filtering processing is further performed on two pairs of mutually adjacent primary differences (d1, d2) and (d2, d3) among the three primary differences d1, d2 and d3 by using a difference filter in a similar manner. Accordingly, the difference between each pair of the primary differences is calculated as secondary differences d4 and d5. The number of combinations of the positive/negative of each of the primary differences and the secondary differences, which are calculated for the four serially adjacent pixels in the image, is 18. FIGS. 6-8 illustrate the relationship between the combinations and the profile shapes of the four pixels. Among the 18 combinations, there are two combinations indicating that there is an edge between the two adjacent pixels G2 and G3, and they are edge 1 and edge 2, which are illustrated in FIG. 6. There are two kinds of edge 1, namely, an edge rising from left to right, which corresponds to the combination (d1, d2, d3, d4, d5)=(+, +, +, +, −), and an edge rising from right to left, which corresponds to the combination (d1, d2, d3, d4, d5)=(−, −, −, −, +). There are four kinds of edge 2, namely, a downwardly convex edge rising from left to right, which corresponds to the combination (d1, d2, d3, d4, d5)=(+, +, +, +, +), an upwardly convex edge rising from left to right, which corresponds to the combination (d1, d2, d3, d4, d5)=(+, +, +, −, −), a downwardly convex edge rising from right to left, which corresponds to the combination (d1, d2, d3, d4, d5)=(−, −, −, +, +), and an upwardly convex edge rising from right to left, which corresponds to the combination (d1, d2, d3, d4, d5)=(−, −, −, −, −). The primary differences d1, d2, and d3 and the secondary differenced d4 and d5 are calculated for four serially adjacent pixels. If the relationship among the positive/negative of each of the differences satisfies the relationship of edge 1 or edge 2, which is illustrated in FIG. 6, it is judged that there is an edge between the two adjacent pixels G2 and G3. If the relationship among the positive/negative of each of the differences satisfies the relationships such as a peak and a trough, which are illustrated in FIGS. 7 and 8, it is judged that there is no edge between the two adjacent pixels G2 and G3. In the following descriptions on the present invention, the edge detection method as described above is referred to as a second edge detection method.
Further, it is also possible to detect whether there is an edge between two adjacent pixels more accurately by combining the first edge detection method and the second edge detection method as described above. Specifically, the primary differences d1, d2, and d3 and the secondary differences d4 and d5 are obtained for the four serially adjacent pixels G1-G4 as in the same manner with the second edge detection method to make a first judgment. In the first judgment, if the relationship among the positive/negative of each of these differences satisfies the relationship of edge 1 or edge 2, which is illustrated in FIG. 6, it is judged that there is an edge between the two adjacent pixels G2 and G3. If the relationship among the positive/negative of each of the differences satisfies the relationship such as a peak or a trough, which is illustrated in FIG. 7, it is judged that there is no edge between the two adjacent pixels G2 and G3. If it is judged that there is an edge between the two adjacent pixels G2 and G3 in the first judgment, judgment is further made as to whether the absolute value of the difference in pixel value between the pixels G2 and G3 is equal to or larger than a predetermined threshold value (second judgment). Only if the second judgment is YES, it is judged that there is a true edge between the pixels G2 and G3. Accordingly, it is possible to prevent a subtle difference in pixel value between the pixels G2 and G3 from being erroneously judged to be an edge when the difference can be regarded as a noise. In the following descriptions on the present invention, the edge detection method as described above is referred to as a third edge detection method.
Further, the first edge detection method may be modified. In the modified edge detection method, if the absolute value of the difference in pixel value between the two adjacent pixels is equal to or larger than a predetermined threshold value (first threshold value), it is judged that there is an edge between the two pixels. In addition, when the absolute value of the difference in pixel value between the two adjacent pixels is less than the first threshold value, if the absolute value of the difference is equal to or larger than a predetermined threshold value (second threshold value), which is less than the first threshold value, and the absolute value of the difference between these two adjacent pixels is the maximum among the absolute values of the differences in pixel value between each pixel pair of mutually adjacent pixels among a plurality of serially adjacent pixels, which is at least four, with these two adjacent pixels at the middle of the plurality of serially adjacent pixels, it may be judged that there is an edge between the two adjacent pixels at the middle of the plurality of serially adjacent pixels. Accordingly, the edge can be detected. In this method, a relatively smooth edge as well as a sharp edge can be detected. In the following descriptions on the present invention, the edge detection method as described above is referred to as a fourth edge detection method.
There are various methods, such as the above examples, for detecting an edge by judging whether there is an edge between pixels in an image. Further, it is also possible to judge the extension direction of the edge (hereinafter referred to as an edge direction, for short) by using these methods. When an interpolation pixel is interpolated between pixels in an image for enlarging or reducing the image, if the pixel value of the interpolation pixel is obtained by selecting a different reference pixel or using a different interpolation operation method depending on whether there is an edge in the vicinity of the interpolation pixel, the direction of the edge, or the like, it is possible to prevent the deterioration in the image quality of the image due to enlargement or reduction. For example, if an edge is not detected between pixels in the vicinity of the interpolation pixel, all the pixels (for example, 16 pixels in a 4×4 pixel block surrounding the interpolation pixel) in the vicinity of the interpolation pixel are used as reference pixels. The pixel value of the interpolation pixel is obtained by performing an interpolation operation by using the pixel values of the reference pixels. In contrast, if an edge is detected between pixels in the vicinity of the interpolation pixel, a judgment is made on which side of the edge the interpolation pixel is located. Further, among the pixels in the vicinity of the interpolation pixel, only the pixels, which are located on the same side of the edge as the interpolation pixel, are used as the reference pixels to obtain the pixel value of the interpolation pixel. Accordingly, the deterioration of the image quality in the enlarged/reduced image can be prevented. Needless to say, besides using the different reference pixels, a different interpolation operation method may be used depending on whether an edge has been detected between the pixels in the vicinity of the interpolation pixel.
In digital images, which are obtained by photoelectrically reading out images recorded on a photography film, or digital images, which are obtained by taking a photograph of a subject with a digital photography device, sampling processing is performed to digitize analog image signals. When the sampling processing is performed, the pixel value of a pixel representing a part, which includes an edge, in a subject (hereinafter referred to as an edge part), is a pixel (hereinafter referred to as an intermediate value pixel) representing an intermediate value between the signal values of the parts on both sides of the edge. FIG. 33B illustrates an example of the pixel composition in a digital image of a subject, which is illustrated in FIG. 33A, in the X direction. In FIG. 33B, a pixel Gb represents a part including an edge L in the subject, and the signal value of the pixel Gb is an intermediate signal value between the signal value of the part on the left side of the edge L and the signal value of the part on the right side of the edge L.
Meanwhile, when an interpolation pixel is interpolated between pixels in an image to enlarge or reduce the image, a pixel located in the vicinity of the interpolation pixel is used as a reference pixel. An interpolation operation is performed by using the pixel value of the reference pixel, and the pixel value of the interpolation pixel is obtained. When the interpolation pixel is interpolated in the vicinity of the pixel Gb during enlargement/reduction processing of the digital image of the subject, which is illustrated in FIG. 33A, if the pixel Gb is used as the reference pixel of the interpolation pixel, the pixel value of the interpolation pixel is influenced by the pixel value of the intermediate value pixel Gb. Therefore, there is a problem that a blur is generated in the enlarged/reduced image. Therefore, it is desirable to judge whether a pixel in the image is the intermediate value pixel to reduce the influence of the intermediate value pixel so that a high quality enlargement/reduction image (enlarged/reduced image) is produced.
Further, as described above, detection processing is performed to detect whether there is an edge between pixels in the vicinity of the interpolation pixel, and if there is an edge, among the pixels located in the vicinity of the interpolation pixel, only the pixels, which are located on the same side of the detected edge as the interpolation pixel, are used as the reference pixels. When the pixel value of the interpolation pixel is obtained by performing an interpolation operation by using the pixel values of the reference pixels, if an intermediate value pixel is included in the reference pixels, the pixel value of the interpolation pixel is influenced by the pixel value of the intermediate value pixel. Therefore, there is a problem that a blur remains in an enlarged/reduced image, and in the worst case, a blur becomes even larger than that of the original image.
Further, in the edge detection method for judging whether there is an edge between adjacent pixels and distinguishing whether there is an edge in a 2×2 pixel block and an edge pattern, only four pixels included in the pixel block are referred to. Therefore, there is a possibility that a noise is detected as the edge.
Further, in the edge detection method as described above, edge detection processing is performed separately for each block. Therefore, an edge in an image is detected as a zigzag line, which is formed by consecutively connecting edges in a plurality of adjacent pixel blocks. Therefore, even a smooth edge in an original image is detected as an edge with a zigzag shape. Meanwhile, as described above, when the image is enlarged or reduced, it is required to perform an interpolation operation based on the position and the direction of the edge in the image to improve the image quality of the enlarged/reduced image. When a noise is detected as an edge, if the interpolation operation is performed based on the detection, the noise remains in the enlarged/reduced image. Therefore, there is a problem that the image quality of the enlarged/reduced image is not good.
If a smooth edge in the original image is detected as an edge with a zigzag shape, when an interpolation operation is performed based on the detection, the shape of an edge in the corresponding part of the obtained image also becomes a zigzag shape. Therefore, there is a problem that the image quality deteriorates.